Collapse source class tables into one SOURCES registry#102
Merged
Conversation
SOURCE_DICT and the two *_SOURCE_PAIRS tables each repeated the same connector
classes, so adding a source meant editing three tables in lockstep (plus the
orchestration list, fixed in the prior change). They are now derived from a
single `SOURCES` registry of `SourceDef(key, site, waterlevel?, analyte?)`:
SOURCE_DICT = {s.key: s.site ...}
WATERLEVEL_SOURCE_PAIRS = {s.key: (s.site, s.waterlevel) for s with waterlevel}
ANALYTE_SOURCE_PAIRS = {s.key: (s.site, s.analyte) for s with analyte}
Adding a source is now one SourceDef entry (plus listing it under the
parameters it serves in PARAMETER_SOURCE_MAP, which stays as authored data —
it encodes which analytes each agency actually reports).
tests/test_source_registry.py ties the registry to PARAMETER_SOURCE_MAP: the
waterlevels agency list must equal the set of sources with a waterlevel class,
and every analyte agency must have an analyte class — so a source wired in one
place but not the other fails a test instead of silently dropping out.
Iteration order of water_level_sources()/analyte_sources() is now source-key
order (was a hand-curated order); full suite (306) confirms nothing depends on
the old order. dg check defs clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This was referenced Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of two config-system improvements. Stacked on #101 (base is
chore/config-validation-dedup) — review/merge that first, then this rebases onto main cleanly.Problem
SOURCE_DICTand the two*_SOURCE_PAIRStables each repeated the same connector classes, so adding a source meant editing three tables in lockstep.Change
All three derive from a single
SOURCESregistry:Adding a source → one
SourceDefentry (plus listing it under the parameters it serves inPARAMETER_SOURCE_MAP, which stays as authored data — it encodes which analytes each agency actually reports, which can't be inferred from class wiring).Desync guard
tests/test_source_registry.pyties the registry toPARAMETER_SOURCE_MAP: the waterlevels agency list must equal the set of sources with a waterlevel class, and every analyte agency must have an analyte class. So a source wired in one place but not the other fails a test instead of silently dropping out of a parameter's source list.Verification
water_level_sources()/analyte_sources()is now source-key order (was hand-curated); 306 passed confirms nothing depended on the old order.dg check defsclean.🤖 Generated with Claude Code